Accessible application cluster topology

ABSTRACT

Some embodiments provide a method for maintaining a cluster topology for a cluster of application instances operating across several datacenters. On a particular machine at which a particular one of the application instances operates, the method maintains a cluster topology that identifies, for each application instance of the cluster, the datacenter in which the application instance operates. From the particular application instance, the method receives a query request for at least a portion of the cluster topology through a programmatic interface. The method provides the requested portion of the cluster topology to the particular application instance. The particular application instance uses the cluster topology for processing application data based on the locations of a set of application instances within the several datacenters.

BACKGROUND

Applications are becoming increasingly complex and are often distributedacross multiple instances. To further exacerbate the already challengingtask of building a distributed system, applications are increasinglybeing deployed across multiple datacenters. As processing requirementsincrease and as processing capabilities become available in differentdatacenters in a hybrid cloud, scaling the distributed applications withcross-datacenter deployment capabilities becomes increasingly difficult.

Some existing solutions to inter-datacenter networking hide theseparation between datacenters, making it difficult to efficientlyprocess data with respect to the different datacenters. Traffic betweendatacenters has higher latency and is more likely to suffer from packetloss than when confined to a single network, so it is desirable to routeand/or batch communications to minimize traffic between datacenters.However, by hiding the separation between datacenters (i.e., treatingeach application instance as though it were operating in a singledatacenter), communication between the application instances of adistributed application becomes increasingly inefficient as theapplication grows.

Alternatively, some distributed applications require thatdatacenter-aware processing be built into the distributed application,requiring significant development resources as each application musttrack the locations of the various application instances and route thedata between the various application instances. Complicated routes mustbe managed and tracked between each pair of individual applicationinstances in a cluster. In order to maintain connectivity, the clustertopology must be continuously monitored in order to update the routesand compensate for partitions, failures and additions.

Some existing solutions provide a more generalized solution, but stillrequire significant management resources as the locations andconnections between the various instances must be managed as instancesare added and removed. For example, in some cases, the distributedapplication requires an admin to create and maintain a VPN across avariety of public cloud infrastructures, leading to significantusability issues. Other alternatives involve treating each datacenter asa mostly-independent application with some special case logic forrouting and/or batching communication between datacenters. However, suchsolutions require significant engineering effort when adding newfeatures.

Some alternatives provide federated management of the variousapplication instances, with centralized control and management for theapplication instances at the different datacenters, but such solutionsoften require significant management resources (e.g., for setup,configuration, and maintenance), or have strict requirements forinterconnectivity of the various application instances. For example,some solutions simplify the problem by requiring that all applicationinstances across all datacenters form a fully connected graph. Thisremoves the need for complicated routing logic, but also preventsdeployment in certain circumstances due to policy or security concerns.As such, a generalized, datacenter-aware, and simplified solution forthe deployment of distributed applications across multiple datacentersis needed.

BRIEF SUMMARY

Some embodiments of the invention provide a cluster manager thatoperates on machines in a network to create a self-managing overlaynetwork that provides both cluster membership and topology, as well asrouting. The overlay network allows distributed applications toreplicate data and segregate functionality based on application instancelocation without needing to directly manage the cluster topology. Theoverlay network also allows cross-datacenter communication to be sent ina way that appears direct irrespective of the underlying configuration.

The self-managing overlay network provides an application-level solutionto the problem of inter-datacenter networking for a cluster ofapplication instances. In some embodiments, the self-managing overlaynetwork is a “datacenter aware” system that provides a programmaticinterface for applications, in order to extend the capabilities ofapplications without requiring customers to develop and maintainelaborate network configurations and without compromising applicationfeatures. Many components can be expanded to communicate acrossdatacenters with minimal effort, and engineers are shielded from thedistraction of the underlying cluster topology while still allowing lowlevel access as necessary.

Some embodiments provide a method for providing access to a clustertopology for a cluster of application instances operating across severaldatacenters. The method of some embodiments provides a distributedcluster manager that tracks application instances in the cluster,tracking to which datacenter each application instance belongs. Thedistributed cluster manager of some embodiments also supports markingcertain application instances (or the machines on which they operate) asaccess points to a cluster—these are the subset of application instancesin each datacenter visible from one or more other datacenters or fromthe public Internet.

The cluster manager, or overlay daemon, of some embodiments providesboth (i) programmatic access to the cluster topology and (ii) an overlaynetwork for routing and forwarding data for application instancesmanaged by the cluster manager. The programmatic access allows theapplication instances to process datacenter-aware data based on thecluster topology, while the overlay network allows the applicationinstances to forward and route data without regard to the currentcluster topology, such that the distributed application does not need tomake routing and forwarding decisions.

In some embodiments, the application instances communicate with thecluster manager to access the cluster topology. The cluster manager fora particular application instance receives a query request for at leasta portion of the cluster topology through a programmatic interfaceprovided by the cluster manager. In some embodiments, the distributedcluster managers each maintain the cluster topology by communicatingwith other cluster managers for the cluster of application instances.

In response to the query request, the cluster manager provides therequested portion of the cluster topology to the particular applicationinstance. The particular application instance then uses the clustertopology for processing data based on the locations of a set ofapplication instances within the plurality of datacenters. Anapplication instance will process data to be sent to a destinationapplication instance differently when destination application instanceis in a different datacenter. For example, when a particular applicationinstance of some embodiments needs to communicate with a remoteapplication instance, the particular application instance gathers a setof data from a set of other application instances within its datacenterand aggregates the data before sending the aggregated data to thedestination application instance, so as to minimize the amount ofcross-datacenter traffic.

The cluster managers, in some embodiments, work together to maintain anup-to-date and consistent cluster topology. In some embodiments, thecluster managers use a gossip protocol to maintain a consistent view ofthe cluster topology across the different datacenters. Each clustermanager periodically communicates with a group of other cluster managers(e.g., a fixed group, a randomized and/or dynamic group, etc.), toupdate and maintain a copy of the cluster topology at each clustermanager.

In addition, the cluster managers of some embodiments provide theability to process data messages through an overlay network for thecluster of application instances operating on machines located acrossseveral datacenters. When the cluster manager receives a data messagefrom a first application instance (operating on the same machine as thecluster manager) and destined for a second application instance in itscluster, the cluster manager determines whether the destinationapplication instance is located in the same or a different datacenter.When the second application instance operates on a second machine in adifferent, second datacenter, the cluster manager encapsulates the datamessage according to an overlay network defined for the cluster ofapplication instances, and forwards the encapsulated data message to apublicly accessible machine of the second datacenter (which may not bethe same machine as the machine on which the destination applicationinstance operates). However, when the second application operates on amachine in the same datacenter as the first machine, the method does notencapsulate the data message, but merely forwards the data message tothe destination machine.

When a new application instance is to be added to a cluster, its clustermanager (i.e., the cluster manager on the machine on which theapplication instance operates) identifies a seed cluster manager thatmanages an application instance that is already a member of the cluster.The cluster manager for the new instance sends a request to be added tothe cluster to the seed cluster manager. The request includesinformation about the new application instance (e.g., address, location,etc.) to be added to the cluster topology.

In some embodiments, the distributed cluster manager adds the newapplication instance to the cluster topology, and constructs efficienttunnels between application instances through a pluggable interface (UDPtunnels, SSH tunnels, VXLAN, STT, etc.). Each application instance isgiven a unique IP within the overlay network. Traffic within a singledatacenter bypasses the overlay network and proceeds with no performancepenalty. Traffic destined for a remote application instance (e.g., anapplication instance in a remote datacenter) is routed to an accesspoint within the remote application instance's datacenter and thenrelayed to the target application instance. Distributed applicationsrunning within the overlay network use overlay IPs for addressing andcan trust that their communications will be routed accordingly.

The seed cluster manager distributes a copy of the cluster topology tothe cluster manager of the new instance. In some embodiments, the seedcluster manager also distributes the copy of the updated clustertopology to other cluster managers for application instances in thecluster (e.g., using a gossip protocol). The cluster manager for the newinstance then processes data for the new application instance based onthe received overlay network (e.g., encapsulating data messages sent bythe new application instance as described above).

The preceding Summary is intended to serve as a brief introduction tosome embodiments of the invention. It is not meant to be an introductionor overview of all inventive subject matter disclosed in this document.The Detailed Description that follows and the Drawings that are referredto in the Detailed Description will further describe the embodimentsdescribed in the Summary as well as other embodiments. Accordingly, tounderstand all the embodiments described by this document, a full reviewof the Summary, Detailed Description, the Drawings and the Claims isneeded. Moreover, the claimed subject matters are not to be limited bythe illustrative details in the Summary, Detailed Description and theDrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth in the appendedclaims. However, for purposes of explanation, several embodiments of theinvention are set forth in the following figures.

FIG. 1 illustrates an example of adding a new application instancewithin a datacenter to a cluster.

FIG. 2 illustrates an example of adding a new application instance fromthe second datacenter to the cluster.

FIG. 3 illustrates an example of adding another second new applicationinstance at the second datacenter to the cluster.

FIG. 4 conceptually illustrates a process of some embodiments forjoining a cluster.

FIG. 5 conceptually illustrates a process of some embodiments forhandling a request from a new application instance to join a cluster ofapplication instances.

FIGS. 6A-B illustrate an example of the distribution of a clustertopology to cluster managers for a cluster of application instancesusing a gossip protocol.

FIG. 7 illustrates an example of an application instance querying acluster manager for a cluster topology.

FIG. 8 conceptually illustrates a process of some embodiments forprocessing data according to a cluster topology.

FIG. 9 illustrates an example of a cluster manager processing a datamessage addressed to an application instance in the same datacenter.

FIG. 10 illustrates an example of a cluster manager processing a datamessage addressed to an application instance in a different datacenterusing the overlay network.

FIG. 11 conceptually illustrates a process of some embodiments forprocessing data messages intercepted by a cluster manager through anoverlay network.

FIG. 12 illustrates an example of communications between applicationinstances of a cluster spread across different datacenters.

FIGS. 13 and 14 illustrate examples of processing received data messagesby the cluster manager at an access point.

FIG. 15 conceptually illustrates a process of some embodiments forprocessing data messages received at an access point from outside thedatacenter of the access point.

FIG. 16 conceptually illustrates a computer system with which someembodiments of the invention are implemented.

DETAILED DESCRIPTION

In the following detailed description of the invention, numerousdetails, examples, and embodiments of the invention are set forth anddescribed. However, it will be clear and apparent to one skilled in theart that the invention is not limited to the embodiments set forth andthat the invention may be practiced without some of the specific detailsand examples discussed.

Some embodiments of the invention provide a cluster manager thatoperates on machines in a network to create a self-managing overlaynetwork that provides both cluster membership and topology, as well asrouting. The overlay network allows distributed applications toreplicate data and segregate functionality based on application instancelocation without needing to directly manage the cluster topology. Theoverlay network also allows cross-datacenter communication to be sent ina way that appears direct regardless of the underlying configuration. Insome embodiments, the cross-datacenter communication refers to varioustypes of data messages that are sent across a cluster of applicationinstances. An overlay network, in some embodiments, is a network that isdecoupled from the underlying physical topology (e.g., asoftware-defined overlay network). The overlay network created by thecluster manager of some embodiments, for example, connects applicationinstances located in several different datacenters.

The data transfer requirements for a distributed application fall intotwo basic categories, datacenter aware and simple. Datacenter awarerequirements require the application instances to identify applicationinstances within the local datacenter and to identify remote applicationinstances so that different routing rules can be applied depending onthe source and destination. Simple data requirements do not require anydatacenter awareness at all. Things like access control, statistics, andconfiguration may be safe and lightweight enough replicate across theentire cluster without any specialized handling. Simple requirementsonly require that the data be transferred between any two applicationinstances, but have no strong performance requirements (e.g., latency,routing, etc.).

The self-managing overlay network provides an application-level solutionto the problem of inter-datacenter networking for a cluster ofapplication instances. In some embodiments, the self-managing overlaynetwork is a “datacenter aware” system that provides a programmaticinterface for applications, in order to extend the capabilities ofapplications without requiring customers to develop and maintainelaborate network configurations and without compromising applicationfeatures. Many components can be expanded to communicate acrossdatacenters with minimal effort, and engineers are shielded from thedistraction of the underlying cluster topology while still allowing lowlevel access as necessary.

Some embodiments provide a method for providing access to a clustertopology for a cluster of application instances operating across severaldatacenters. The method of some embodiments provides a distributedcluster manager that tracks application instances in the cluster,tracking to which datacenter each application instance belongs. Thedistributed cluster manager of some embodiments also supports markingcertain application instances (or the machines on which they operate) asaccess points to a cluster—these are the subset of application instancesin each datacenter visible from one or more other datacenters or fromthe public Internet.

The cluster manager, or overlay daemon, of some embodiments providesboth (i) programmatic access to the cluster topology and (ii) an overlaynetwork for routing and forwarding data for application instancesmanaged by the cluster manager. The programmatic access allows theapplication instances to process datacenter-aware data based on thecluster topology, while the overlay network allows the applicationinstances to forward and route data without regard to the currentcluster topology, such that the distributed application does not need tomake routing and forwarding decisions.

In some embodiments, the application instances communicate with thecluster manager to access the cluster topology. The cluster manager fora particular application instance receives a query request for at leasta portion of the cluster topology through a programmatic interfaceprovided by the cluster manager. In some embodiments, the distributedcluster managers each maintain the cluster topology by communicatingwith other cluster managers for the cluster of application instances.

In response to the query request, the cluster manager provides therequested portion of the cluster topology to the particular applicationinstance. The particular application instance then uses the clustertopology for processing data based on the locations of a set ofapplication instances within the plurality of datacenters. Anapplication instance will process data to be sent to a destinationapplication instance differently when destination application instanceis in a different datacenter. For example, when a particular applicationinstance of some embodiments needs to communicate with a remoteapplication instance, the particular application instance gathers a setof data from a set of other application instances within its datacenterand aggregates the data before sending the aggregated data to thedestination application instance, so as to minimize the amount ofcross-datacenter traffic.

The cluster managers, in some embodiments, work together to maintain anup-to-date and consistent cluster topology. In some embodiments, thecluster managers use a gossip protocol to maintain a consistent view ofthe cluster topology across the different datacenters. Each clustermanager periodically communicates with a group of other cluster managers(e.g., a fixed group, a randomized and/or dynamic group, etc.), toupdate and maintain a copy of the cluster topology at each clustermanager.

In addition, the cluster managers of some embodiments provide theability to process data messages through an overlay network for thecluster of application instances operating on machines located acrossseveral datacenters. When the cluster manager receives a data messagefrom a first application instance (operating on the same machine as thecluster manager) and destined for a second application instance in itscluster, the cluster manager determines whether the destinationapplication instance is located in the same or a different datacenter.When the second application instance operates on a second machine in adifferent, second datacenter, the cluster manager encapsulates the datamessage according to an overlay network defined for the cluster ofapplication instances, and forwards the encapsulated data message to apublicly accessible machine of the second datacenter (which may not bethe same machine as the machine on which the destination applicationinstance operates). However, when the second application operates on amachine in the same datacenter as the first machine, the method does notencapsulate the data message, but merely forwards the data message tothe destination machine. In this application, reference is made to datamessages and/or packets, but it should be understood that these terms,as used herein, may refer to various formatted collections of bits thatmay be sent across a network, such as Ethernet frames, IP packets, TCPsegments, UDP datagrams, etc.

When a new application instance is to be added to a cluster, its clustermanager (i.e., the cluster manager on the machine on which theapplication instance operates) identifies a seed cluster manager thatmanages an application instance that is already a member of the cluster.The cluster manager for the new instance sends a request to be added tothe cluster to the seed cluster manager. The request includesinformation about the new application instance (e.g., address, location,etc.) to be added to the cluster topology.

In some embodiments, the distributed cluster manager adds the newapplication instance to the cluster topology, and constructs efficienttunnels between application instances through a pluggable interface (UDPtunnels, SSH tunnels, VXLAN, STT, etc.). Each application instance isgiven a unique IP within the overlay network. Traffic within a singledatacenter bypasses the overlay network and proceeds with no performancepenalty. Traffic destined for a remote application instance (e.g., anapplication instance in a remote datacenter) is routed to an accesspoint within the remote application instance's datacenter and thenrelayed to the target application instance. Distributed applicationsrunning within the overlay network use overlay IPs for addressing andcan trust that their communications will be routed accordingly.

The seed cluster manager distributes a copy of the cluster topology tothe cluster manager of the new instance. In some embodiments, the seedcluster manager also distributes the copy of the updated clustertopology to other cluster managers for application instances in thecluster (e.g., using a gossip protocol). The cluster manager for the newinstance then processes data for the new application instance based onthe received overlay network (e.g., encapsulating data messages sent bythe new application instance as described above).

The above description introduces a self-managing overlay network.Several more detailed embodiments are described below. Section Idescribes examples of building up a self-managing overlay network.Section II describes the use and distribution of a cluster topology fora cluster of application instances. Section III describes the use of theoverlay network to forward and route data between application instancesof the cluster. Finally, Section IV describes an electronic system withwhich some embodiments of the invention are implemented.

I. Cluster Set-Up

As mentioned, when a new application instance is to be added to acluster, its cluster manager (i.e., the cluster manager on the machineon which the application instance operates) identifies a seed clustermanager that manages an application instance that is already a member ofthe cluster. The cluster manager for the new instance sends a request tobe added to the cluster to the seed cluster manager. The requestincludes information about the new application instance (e.g., address,location, etc.) to be added to the cluster topology.

In some embodiments, the distributed cluster manager adds the newapplication instance to the cluster topology, and constructs efficienttunnels between application instances. The seed cluster manager alsodistributes a copy of the cluster topology to the cluster manager of thenew instance. In some embodiments, the seed cluster manager alsodistributes the copy of the updated cluster topology to other clustermanagers for application instances in the cluster (e.g., using a gossipprotocol). The cluster manager for the new instance then processes datafor the new application instance based on the received overlay network(e.g., encapsulating data messages sent by the new application instanceas described above).

FIG. 1 illustrates an example of adding a new application instancewithin a datacenter to a cluster in three stages 101-103. This figureshows two datacenters 120 and 125, which are connected by a network (notshown) such as the public Internet, a private network, etc. Thedifferent datacenters of some embodiments include multiple datacentersowned and/or operated by a particular entity (e.g., remote datacentersfor a corporate enterprise network, etc.). In some embodiments, thedatacenters are operated by various unrelated entities (e.g., cloudhosting datacenters operated by various vendors, enterprise networkdatacenters, etc.). Although only two datacenters are shown in theexamples of the figures of this application, it should be evident to oneskilled in the art that the methods and systems are applicable to anynumber of different datacenters.

The first stage 101 shows three application instances AI1-AI3 withcorresponding cluster managers CM1-CM3 operating at datacenter 120. AI1and AI2 are shown as being a part of a cluster of application instances,as indicated by the solid lines. Application instance AI3 is not yet apart of the cluster, as indicated by the dashed line.

A cluster is a related group of application instances of a distributedapplication, that operate together to perform the functions of thedistributed application. In some embodiments, the cluster includesapplication instances of different types. For example, the cluster ofsome embodiments includes multiple different distributed applicationsthat operate together (e.g., sharing and replicating data, distributedprocessing, etc.). In some embodiments, all of the application instancesare of the same distributed application. In some embodiments, thedifferent application instances are the same application, but dividedinto groups for performing different functions of the application.

In the example of FIG. 1, each application instance of the cluster ofapplication instances operates on a same machine as a cluster manager ofthe plurality of cluster managers, and each cluster manager manages asingle application instance on its particular machine. In someembodiments, the cluster manager is a separate service from theapplication instance (e.g., operating in the background of the machineas a daemon), while in other embodiments, the cluster manager is amodule that operates as a part of an application instance. In someembodiments, a single cluster manager manages multiple applicationinstances. For example, in some embodiments, each application instanceoperates within a container and multiple containers operate on a singlemachine. The single machine of some embodiments has a cluster managerthat manage the multiple application instances.

Although the application instances are shown as individual elements,each application instance operates on a machine (not shown) in thedatacenter 120. In some cases, each application operates on its ownmachine, while in other cases, multiple application instances of acluster run on a single machine. In some embodiments, the machines onwhich the application instances operate are virtual machines, orcontainers, which provide isolated environments within which anapplication instance (or multiple application instances) an operate. Thecluster managers may operate in the same virtual machine as theapplication instance in some embodiments, while in other embodiments thecluster managers operate on the same physical machine as theirrespective application instance but outside the virtual machine orcontainer in which the application instance operates. In someembodiments, the cluster manager operates on its own virtual machine.

In addition, application instance AI1 is shown to be an access point forthe datacenter 120. An access point for a datacenter provides a publiclyaccessible address (e.g., a Uniform Resource Locator (URL), a public IPaddress, etc.), which can be accessed by machines in other datacentersand networks. The access point may refer to the machine itself, or to acluster manager or application instance that operates on the machine.Non-access points in a datacenter may not be reached directly fromoutside of the datacenter.

In the second stage 102, the cluster manager CM3 for the applicationinstance AI3 sends a cluster joining request to CM2. In this example,the cluster manager CM2 for application instance AI2 is a seed clustermanager. A seed cluster manager, in some embodiments, is a clustermanager specifically made available for adding new application instancesto a cluster. In some embodiments, all of the cluster managers have theability to act as a seed cluster manager, and the configuration for anew application instance identifies the seed cluster manager that willbe used.

In some embodiments, the cluster manager for application instance AI3identifies the seed CM as an address (e.g., for the application instanceAI2) stored in the configuration data for its local application instanceAI3. The configuration (e.g., a user configuration, an automaticallygenerated configuration, etc.) for a new application instance mayinclude a single seed cluster manager (or application instance address)or more than one seed cluster manager in some embodiments. For instance,the configuration might include several seed cluster managers in thesame datacenter as the new application instance, or one or more clustermanagers located in another datacenter.

The request 150 includes information from the requesting applicationinstance AI3 to join a cluster of application instances. The request 150of some embodiments includes an address (IP3), the applicationinstance's availability as an access point (Access), and the datacenter(DC1) in which the application instance is located. In otherembodiments, the request may include additional fields, such as forauthentication, role assignment, etc. In some embodiments, the addressstored in the request is a desired address in the overlay network.Alternatively, or conjunctively, the request stores a local address forthe new application instance, assigned for the application instancewithin the datacenter. In some embodiments, when the applicationinstance is an access point, the request also includes a publiclyaccessible address (e.g., a URL) for the application instance. Theaddress of the application, in some embodiments, is an address of themachine (e.g., the virtual machine or physical machine) on which theapplication instance operates.

In some embodiments, the request 150 is sent to the seed cluster managerCM2 as a control message, to distinguish the request from other trafficreceived by the cluster manager. The request 150 of some embodiments isidentified as a control message when an address of the data messageincludes an address that is invalid for a particular data messageprotocol (e.g., TCP, UDP).

The third stage 103 shows that application instance AI2 responds to therequest 150 with a response 155. The response 155 of some embodimentsincludes the cluster topology for the cluster of application instances.The cluster topology in the response 155 has been updated by applicationinstance AI2 to include the new application instance AI3, indicatingapplication instance AI3's address, accessibility, and datacenterlocation. In some embodiments, the seed cluster manager for AI2 alsosets up tunnels for the new application instance AI3 to be used for anoverlay network that is set up for the cluster of application instances.In other embodiments, the cluster manager CM3 sets up its tunnels basedon the cluster topology information, which includes the networkaddresses for the various other application instances in the cluster.

In the example of FIG. 1, a new application instance joined a cluster ofapplication instances from within the same datacenter. As a clustergrows and expands to additional datacenters, it becomes important for anew application instance to be able to join the cluster from outside ofa datacenter. When external application instances (e.g., applicationinstances running in other datacenters or networks) need to be added toa network cluster, the cluster manager of the external applicationinstance can communicate with a cluster manager at an access point(i.e., a cluster manager with a publicly accessible address). In someembodiments, the cluster manager at the access point is used as a seedcluster manager for the new application instance.

FIG. 2 illustrates an example of adding a new application instance fromthe second datacenter 125 to the cluster in three stages 201-203. Thefirst stage 201 shows the same datacenters 120 and 125 with applicationinstances AI1-AI3, as at this point AI3 has successfully joined thecluster. In addition, the first stage 201 shows a new applicationinstance AI4, located in datacenter 125 (shown as dashed because thisapplication instance has not yet joined the cluster). As no otherapplication instances of the cluster are currently operating indatacenter 125, the cluster manager CM4 for application instance AI4must communicate with a cluster manager from datacenter 120 in order tojoin the cluster. Because the application instance AI4 operates outsideof datacenter 120, some embodiments require that its specified seedcluster manager be one of the cluster managers operating as an accesspoint within the datacenter 120. The access points (e.g., AI3 and AI1)for a cluster within a particular datacenter allow application instancesthat operate outside of the particular datacenter to communicate withthe application instances within the particular datacenter, withoutforcing all of the application instances and the machines on which theyoperate to be publicly accessible. In this case, the seed clustermanager specified by the configuration for AI4 is the cluster managerCM1.

The second stage 202 shows that the cluster manager CM4 for theapplication instance AI4 sends a request 250 to the seed cluster managerCM1. The request 250 includes the address (IP4), access status (Access),and DC location (DC2) for the new application instance AI4. While inthis example, the access point cluster manager is also the seed clustermanager specified by the configuration for application instance AI4, insome embodiments the seed cluster manager for a new application instancemay be located in another datacenter but also is not an access point. Insuch cases, the cluster manager for the new application instance sendsits request to the access point for the datacenter of the seed clustermanager, which forwards the request to the seed cluster manager. Inother embodiments, the seed cluster manager for a new applicationinstance is required to be either in the same datacenter as the newapplication instance or located at an access point.

Finally, in the third stage 203, the seed cluster manager CM1 respondsto the request 250 with a response 255. The response 255 includes theupdated cluster topology, which has been updated to include the newapplication instance AI4. This cluster topology, in some embodiments,specifies for each application instance in the cluster, the network (IP)address, the datacenter in which the application instance operates, andwhether or not the application instance operates on an access node forthe cluster. The third stage 203 also shows that application instanceAI3 has also been designated as an access point for the cluster.

FIG. 3 illustrates an example of adding another new application instanceat the second datacenter to the same cluster in three stages 301-303. Inthis example, a second application instance AI5 at datacenter 125 is tobe added to the cluster of application instances. As in the examplesabove, the first stage 301 shows the datacenters 120 and 125, withcluster application instances AI1-AI4. A new application instance AI5 tobe added to the cluster is shown at datacenter 125, as indicated by thedashed lines.

The second stage 302 shows that the cluster manager CM5 for the newapplication instance AI5 sends a request 350 to its seed cluster managerCM4, which is located in the same datacenter. The request indicates thatthe new application instance AI5 has an address (IP5), is not publiclyaccessible (NA), and is located in datacenter 125 (DC2). In the thirdstage 303, application instance AI4 sends back a response 355 with anupdated cluster topology, including the addresses, access statuses, anddatacenter locations for each application instance AI1-AI5. The thirdstage 303 also shows that application instance AI5 has been added to thecluster.

As the application instances and cluster managers may operate on virtualmachines, these virtual machines (and thus the application instance) maymigrate within a datacenter (or in rare cases, from one datacenter toanother). When a VM migrates within a datacenter, assuming it retainsits existing network addresses, the cluster topology is not affected. Ifa VM migrates to a different datacenter or changes its network address,then some embodiments require the application instance to be removedfrom the cluster and rejoin.

FIG. 4 conceptually illustrates a process 400 of some embodiments forjoining a cluster (i.e., for adding a local application instance to acluster). The process 400 is performed in some embodiments by a clustermanager for a new application instance that starts up on a machine onwhich the cluster manager runs, using configuration data provided withthe new application instance.

As shown, the process 400 identifies (at 405) a seed instance for thecluster. In some embodiments, the cluster manager identifies the seedinstance based on the configuration (e.g., a user configuration, anauto-generated configuration, etc.) for the new application instance. Insome embodiments, the configuration for a new application instance isrequired to specify at least one seed instance. This may be a seedcluster manager, or a seed address (e.g., IP address) used by anapplication instance (to which a seed cluster manager corresponds).

Once the seed instance has been identified, the process 400 sends arequest to join the cluster to the identified seed instance. The requestprovides information about the new application instance to the seedcluster manager. In some embodiments, the request includes a set ofaddresses for the new application instance (e.g., an overlay networkaddress, a local address within the datacenter, etc.), the applicationinstance's availability as an access point (i.e., whether or not itoperates as an access point for application instances outside of itsdatacenter), and the datacenter in which the application instance islocated. In other embodiments, the request may include additionalfields, such as fields for authentication, role assignment, etc., whichmay be used by the application instances of the cluster.

The process 400 then receives (at 415) the cluster topology for thecluster, which now includes the new application instance. The clustertopology of some embodiments identifies a datacenter for eachapplication instance in the cluster (i.e., the datacenter in which eachapplication instance operates). In some embodiments, the clustertopology also includes overlay network data for the cluster. The overlaynetwork data of some embodiments describes an overlay network thatconnects the application instances of the cluster with each other. Thecluster topology and the overlay network will be described in furtherdetail below. In addition, as described below, the cluster topology maynot be complete, if the seed cluster manager does not have the mostup-to-date view of the network. However, through the gossip processdescribed below, both of the cluster managers will quickly learn thefull scope of the cluster topology.

The new cluster manager (and new application instance, via the clustermanager) is then able to use the received cluster topology to processdata based on the cluster topology, while forwarding data through theoverlay network without regard for the cluster topology (e.g.,irrespective of the datacenter in which a destination applicationinstance is located).

FIG. 5 conceptually illustrates a process 500 of some embodiments forhandling a request from a new application instance (e.g., from a clustermanager for a new application instance) to join a cluster of applicationinstances. The process 500 is performed by a cluster manager identifiedas a seed instance in some embodiments by the new application instance(i.e., by the configuration of the new application instance).

As shown, the process 500 receives (at 505) a request to join thecluster. In some embodiments, the request provides information about thenew application instance to the cluster manager performing the process500. In some embodiments, the request includes a set of addresses forthe new application instance (e.g., an overlay network address, a localaddress within the datacenter, etc.), the new application instance'savailability as an access point (i.e., whether or not it operates as anaccess point for application instances outside of its datacenter), andthe datacenter in which the application instance is located. In otherembodiments, the request may include additional fields, such as fieldsfor authentication, role assignment, etc., which may be used by theapplication instances of the cluster.

The process 500 then determines (at 510) whether the applicationinstance is allowed to join the cluster. In some cases, a cluster mayinclude security or other restrictions that specify requirements forallowing access to the cluster. For example, a request to join a clustermight be required to include a correct signature (e.g., HMAC, etc.), beencrypted with a correct key, or otherwise provide proof of acryptographic secret. If no proof (signature, encrypted message, etc.)is provided, then the request will be denied. In other cases, when thereis no security, the request should always be allowed. When the newapplication instance is not allowed to join the cluster (e.g., becausethe request does not meet the security requirements for joining thecluster), the process 500 ends. Some embodiments additionally send arejection message to the requesting cluster manager.

When the application instance is allowed to join the cluster, theprocess 500 updates (at 515) a cluster topology to reflect this newapplication instance. In some embodiments, a cluster manager for eachapplication instance maintains the cluster topology that reflectslocations (e.g., addresses and datacenter locations) for each of theapplication instances of the cluster. The new application instance isadded to this cluster topology, using the information contained in therequest (e.g., the datacenter, address, and access point status for thenew application).

The process 500 then sends (at 520) the updated cluster topology to therequesting cluster manager (i.e., the cluster manager for the newapplication instance). This allows the requesting cluster manager tosend packets for the application instance according to the overlaynetwork defined based on the cluster topology, as well as to provide thecluster topology to the application instance as needed. The process 500then ends.

II. Cluster Topology

The cluster topology maintained by the cluster managers enables thecluster managers to provide their respective application instances withinformation about the other application instances in the cluster. Theapplication instances can then process data and communications based onthe cluster topology, allowing distributed applications to optimizeperformance by minimizing the distribution of data across datacenters.The cluster managers of some embodiments share the cluster topology witheach other and provide a query interface for the application instances.

As cluster managers for the different application instances update thecluster topology (e.g., as application instances are added and removedfrom the cluster), the cluster managers need to communicate anddistribute the changes with each other, so that each cluster manager canmaintain a current view of the cluster topology. In some embodiments,the cluster managers for the application instances of the clusterdistribute the cluster topology among cluster managers operating acrossseveral datacenters.

In some embodiments, the cluster managers broadcast the cluster topologyto all the other cluster managers. Alternatively, the cluster managersof some embodiments only broadcast the cluster topology to the othercluster managers within the same datacenter and to a single access pointin each of the other datacenters.

In some embodiments, rather than broadcasting the cluster topology, thecluster managers use a gossip protocol to distribute the topology,reducing the amount of communications necessary to reach a consistentview of the topology across the cluster. The cluster managersperiodically select one or more other cluster managers of the cluster towhich to send their current view of the cluster topology. In someembodiments, the other cluster managers are selected randomly, while inother embodiments each cluster manager has a specific set of clustermanagers to which it sends its view of the cluster topology. In someembodiments, each cluster manager only sends its view of the clustertopology to the other cluster managers in the same datacenter and toaccess points in other datacenters. In other embodiments, however, eachcluster manager may send its view of the cluster topology to any clustermanager for an application instance in the cluster.

FIGS. 6A-B illustrate an example of the distribution of a clustertopology to cluster managers for a cluster of application instancesusing a gossip protocol in four stages 601-604. The first stage 601shows five cluster managers CM1-CM5, that each store respective versionsof the cluster topology 605. Each of the cluster managers CM1-CM5 is formanaging one or more application instances (not shown) of a cluster.

This example follows from the example of FIG. 3 and shows, for eachcluster manager CM1-CM5, the cluster topology maintained by each clustermanager, with application instances represented as boxes. Each group ofboxes represents one of the datacenters 120 and 125. More specifically,the figure shows that cluster managers CM1-CM3 reflect outdated versionsof the cluster topology 605, with three application instances (i.e.,AI1-AI3) in the first datacenter, but only one application instance(i.e., AI4) in the second datacenter. As shown in FIG. 3, clustermanagers CM4 and CM5 were updated with the most recent cluster topologywhen application instance AI5 joined the cluster through applicationinstance AI4.

In the first stage 601, cluster manager CM4 sends a control message 650with its cluster topology to cluster manager CM2. In some embodiments,the cluster manager CM4 sends this control message 650 with an invalidaddress (e.g., an address that is invalid for a particular data messageprotocol (e.g., TCP, UDP)) in order to identify the message as a controlmessage, rather than a normal data message sent by an applicationinstance that is routed between the cluster managers for the applicationinstances. The receiving cluster manager CM2 of some embodiments parsesthe data message to retrieve the cluster topology.

The second stage 602 shows that CM2 has been updated with the currentversion of the cluster topology 605. The second stage 602 also showsthat CM1 (which does not yet have the most updated cluster topology)sends a control packet 655 to CM5. In some embodiments, the clustermanagers send updates at specific intervals or at random intervals.Alternatively, or conjunctively, a particular cluster manager sends thecluster topology to other cluster managers whenever the cluster topologyat the particular cluster manager changes.

In the third stage 603, CM5 has not updated its cluster topology 605with the cluster topology data received from CM1, because the versionstored at CM5 is more current and complete than the one received fromCM1. To determine whether a received topology is up-to-date, someembodiments use timestamps for the most recent change to their view ofthe topology and add these timestamps to the control message. Otherembodiments use checks such as ensuring that the most recently addedapplication instance stored in the local view of the cluster topology isin the received topology view, and rejecting the received view if thisis not the case. In addition, the third stage 603 shows that CM5 sends aresponse packet 660 with the current cluster topology back to CM1. Thecluster managers of some embodiments send such a response when anout-of-date topology view is received.

The fourth stage 604 illustrates that CM1 has been updated with the mostcurrent version of the cluster topology and that CM2, which waspreviously updated with the current version of the cluster topology,sends an update control message to CM3. In this way, all of the clustermanagers are updated with the current cluster topology.

In addition to updating the cluster topology with new applicationinstances and based on received cluster topology data from other clustermanagers, the cluster managers of some embodiments further monitorexisting application instances in order to maintain the clustertopology. In some embodiments, a cluster manager detects when anapplication instance that it manages is no longer available, removes theapplication instance from the cluster topology, and distributes theupdated cluster topology without the removed application instance.

As mentioned, the application instances communicate with the clustermanagers of some embodiments to access the cluster topology. The clustermanager for a particular application instance receives a query requestfor at least a portion of the cluster topology through a programmaticinterface provided by the cluster manager. In some embodiments, thedistributed cluster managers each maintain the cluster topology bycommunicating with other cluster managers for the cluster of applicationinstances.

In response to the query request, the cluster manager provides therequested portion of the cluster topology to the particular applicationinstance. The particular application instance then uses the clustertopology for processing data based on the locations of a set ofapplication instances within the plurality of datacenters. Anapplication instance will process data to be sent to a destinationapplication instance differently when destination application instanceis in a different datacenter. For example, when a particular applicationinstance of some embodiments needs to communicate with a remoteapplication instance, the particular application instance gathers a setof data from a set of other application instances within its datacenterand aggregates the data before sending the aggregated data to thedestination application instance, so as to minimize the amount ofcross-datacenter traffic.

FIG. 7 illustrates an example of an application instance querying acluster manager for a cluster topology in three stages 701-703. Theexample of this figure shows a machine 720. The machine 720 of someembodiments is a virtual machine operating on a host machine, or couldbe a host machine itself (e.g., with the applications operating incontainers or in separate virtual machines, and the cluster manageroperating in a separate container or virtual machine or in a hostoperating system of the machine). The machine 720 includes a clustermanager 722, an interface 724 for communicating with the network (notshown), and application instances AI1 and AI2, which operate on themachine 720. The cluster manager 722 includes a query engine 732, astorage 734, a packet processor 736, forwarding tables 738, and anencapsulator 790.

In this example, a single cluster manager 722 manages multipleapplication instances AI1 and AI2. These may be two applicationinstances belonging to the same distributed application cluster, or twoapplication instances of separate clusters (of the same application orof completely different applications). That is, a single cluster manageroperating on a machine may manage multiple clusters in some embodiments.

The first stage 701 shows that application instance AI1 sends a queryrequest 750 to the query engine 732. Though not shown, in someembodiments the cluster manager 722 includes a programmatic interface(e.g., an application programming interface (API)) through which theapplications can query the cluster manager. Some embodiments exposevarious query commands that allow the application to specify the portionof the cluster topology in which they are interested (e.g., the entiretopology, only the local datacenter, only a particular datacenter,etc.).

The query engine 732 queries the storage 734 for data regarding thenetwork overlay or the cluster topology stored in the storage 734 of thecluster manager 722. The queries sent by the application instance AI1 ofsome embodiments may include requests for a datacenter associated with asecond application instance, for a complete list of the applicationinstances of the cluster and their associated datacenters, and/or for alist of application instances in a particular datacenter.

In the second stage 702, the application instance AI1 receives aresponse 755 from the query engine 732 of the cluster manager 722,containing the requested portion of the cluster topology. The thirdstage 703 shows that application instance AI1 stores the receivedcluster topology in a local storage of the application instance AIL Theapplication instance AI1 uses the cluster topology, which provideslocation data (e.g., addresses and datacenter locations), in order tointelligently process data between the different datacenters. In thisexample, the application instance AI1 retrieves the cluster topology andstores it for later use, but in some embodiments, the applicationinstance AI1 communicates through a programmatic interface with thequery engine 732 of the cluster manager 722 on an as-needed basis. Theprogrammatic interface provides a standardized interface for developersto use to access the cluster topology and to forward data using anoverlay network.

FIG. 8 conceptually illustrates a process 800 of some embodiments forprocessing data according to a cluster topology. This process 800 isperformed by an application instance that queries its associated clustermanager. The process is exemplary of a variety of data-processingoperations that an application instance might perform using the clustertopology information from its local cluster manager. In this example,the application instance uses the cluster topology to process data in adatacenter-aware manner, as one instance of an application that spansmultiple datacenters (e.g., a machine or network logging application).

As shown, the process 800 begins by querying (at 805) the clustermanager to retrieve at least a portion of the cluster topology. Asmentioned above, the application may, in some embodiments, request theentire cluster topology, the location of a particular applicationinstance in the cluster, the list of application instances in aparticular datacenter (either its local datacenter or a remotedatacenter), etc. The process then receives (at 810) the requestedcluster topology information from the cluster manager through thecluster manager programmatic interface.

Using the received network topology (either immediately after receivingthe network topology, or a duration thereafter), the process determines(at 815) whether a destination for a particular application-relatedcommunication of data is located in the same datacenter as theapplication instance performing the process, or at a differentdatacenter. For instance, the application might only want to send onecommunication across datacenters, rather than having each applicationinstance at a first datacenter send each of their data separately to adestination application instance at a second datacenter. As an example,a machine logging and analytics application might need to aggregate logdata for numerous virtual machines distributed across numerousdatacenters. Ideally, the amount of cross-datacenter traffic should beminimized, so one application instance in each datacenter shouldaggregate the data for its local datacenter, and then send the data to adestination in another datacenter.

Thus, when the process determines that the destination for theapplication-related data is in its own datacenter, the process forwards(at 825) the data to the destination application instance. Otherwise, ifthe destination is in a remote datacenter (i.e., not the same datacenteras the current application instance), then the process aggregates (at820) the application data for the datacenter. In some embodiments, theprocess 800 aggregates this data by collecting query results fromseveral other application instances operating in the same datacenter andprocessing the data to generate streamlined or summarized results to thedestination application instance, reducing the amount of data and delaythrough the network. The process 800 then forwards (at 825) theaggregated data to the destination application instance. When thedestination is in a different datacenter, the process 800 of someembodiments forwards the data to an access point in the otherdatacenter, which then forwards the aggregated data to the destination(if the destination is not at the access point).

This process assumes that the application instance performing theprocess 800 is an aggregation point for the distributed applicationwithin its datacenter. In other cases, the application instance forwardsits application-related data to a local aggregation point applicationinstance, at which juncture it need not communicate with the eventualdestination application instance for the data.

III. Overlay Network

The cluster managers of some embodiments provide an overlay network thatallows application instances to seamlessly connect as though running ina single datacenter. In some embodiments, the application instances inthe cluster form a complete graph in the overlay network. In someembodiments, every application instance is connected to every otherapplication instance through the overlay network, even when only oneapplication instance in each datacenter is exposed publicly. The clustermanagers allow the distributed application to be deployed in anotherwise unsupported configuration with no change to the core codebase.Because routing is handled within the networking stack, no applicationchanges are required for immediate visibility of remote applicationinstances.

The cluster managers of some embodiments only process packets for remotedatacenters through the overlay network for improved performance. Whenthe cluster manager receives a data message from a first applicationinstance (operating on the same machine as the cluster manager) anddestined for a second application instance in its cluster, the clustermanager determines whether the destination application instance islocated in the same or a different datacenter. When the secondapplication instance operates on a second machine in a different, seconddatacenter, the cluster manager encapsulates the data message accordingto an overlay network defined for the cluster of application instances,and forwards the encapsulated data message to a publicly accessiblemachine of the second datacenter (which may not be the same machine asthe machine on which the destination application instance operates).However, when the second application operates on a machine in the samedatacenter as the first machine, the method does not encapsulate thedata message, but merely forwards the data message to the destinationmachine.

FIG. 9 illustrates an example of a cluster manager processing a datamessage addressed to an application instance in the same datacenter overtwo stages 901 and 902. The first stage 901 shows machine 720 asdescribed above with reference to FIG. 7. In the first stage 901,application instance AI1 sends a data message 950 with a destinationaddress of application instance AI3 to the packet processor 736 of thecluster manager 722. In some embodiments, the application instancesdirect their data messages to interface 724, but packet processor 736intercepts all data messages coming from the distributed applicationinstances AI1 and AI2. If AI2 was a non-distributed application, in someembodiments, the cluster manager would not intercept its data messages,instead allowing them to proceed directly to the interface.

The first stage 901 also shows that the packet processor 736 accesses aset of forwarding tables 738 to determine how to forward the datamessage through the network. In some embodiments, the cluster managerupdates the forwarding tables 738 based on the cluster topology toencapsulate data messages destined for remote datacenters. The clustermanagers of some embodiments maintain a map of shortest paths to eachother member of the cluster (i.e., identifying the optimal access pointin the remote datacenter for each remote application instance). In someembodiments, the cluster manager uses a first network interface fornon-cluster communication and creates a second network interface forcommunicating through the overlay network. In some such embodiments, thenetwork address for the second network interface is assigned based on aconfiguration for the application instance.

In this example, the data message 950 is destined for AI3, whichoperates on a different machine in the same datacenter. The second stage902 shows that data message 950 bypasses the encapsulator 740 and isforwarded through interface 724 to AI3, based on the information in theforwarding tables 738 that specifies that AI3 is located in the samedatacenter as the cluster manager 722.

In some embodiments, the encapsulator 740 is for encapsulating data forremote application instances and forwarding the data to other clustermanagers of the cluster. The forwarding tables of some embodimentsforward any data messages addressed to application instances in the samedatacenter directly, without encapsulating the data messages. Thisprevents any impact to traffic for application instances in the samedatacenter (i.e., it alleviates the send- and receive-side processingassociated with encapsulations), as well as other traffic unrelated tothe cluster.

FIG. 10 illustrates an example of a cluster manager processing a datamessage addressed to an application instance in a different datacenterusing the overlay network in two stages 1001-1002. As in the example ofFIG. 9, the first stage 1001 shows that data message 1050 is processedby the forwarding tables 738 (e.g., after being intercepted by thepacket processor 736). In this case, the data message is sent by AI2,rather than AI1. In some embodiments, AI2 is an application instance inthe same cluster as AI1, in which case the same set of forwarding tablesand overlay topology is used. However, in other embodiments, applicationinstance A2 belongs to a completely separate application cluster, inwhich case the cluster manager uses a separate cluster topology and setof forwarding tables.

The second stage 1002 shows that, because the data message 1050 has adestination address for an application instance located in a remotedatacenter, the encapsulator 740 encapsulates the data message 1050 withan overlay network encapsulation (including, e.g., a destination IPaddress used by destination in the overlay network) and sends the datamessage according to the forwarding tables (e.g., to an access point inthe remote datacenter. The encapsulation of some embodiments is based ona tunneling protocol, such as User Datagram Protocol (UDP) tunneling,Generic Routing Encapsulation (GRE) tunneling, Virtual Extensible LAN(VXLAN) tunneling, Stateless Transport Tunneling (STT), Secure Shell(SSH) tunneling, etc.).

In some embodiments, when the machine 720 is a VM or other data computenode that operates on top of virtualization software (e.g., ahypervisor, virtual machine monitor, etc.) with a software forwardingelement (e.g., a virtual switch), the data message 950 or 1050 is sentto the virtualization software forwarding element. The virtualiationsoftware forwarding element handles data messages from the machine 720as it would with any other data messages in some embodiments, sendingthese data messages out onto the physical network of the datacenter.

FIG. 11 conceptually illustrates a process of some embodiments forprocessing data messages intercepted by a cluster manager through anoverlay network. The process is performed by a cluster manager thatoperates on a machine with an application instance belonging to adistributed application cluster in some embodiments, whenever theapplication instance sends a data message to another applicationinstance in its cluster.

As shown, the process 1100 intercepts (at 1105) a data message from asource application instance located on its machine. In some embodiments,when an application instance that belongs to a cluster managed by thecluster manager sends a data message to an application instance in itscluster, the cluster manager automatically intercepts the data message.When the application instance either does not belong to a cluster, or issending traffic outside of the cluster, the cluster manager operating onthe machine does not intercept such data messages.

The process 1100 then determines (at 1110) whether the data message isaddressed to an application instance in the same datacenter. Asindicated, at this point the cluster manager has already determined thatthe data message is addressed to another application instance in thecluster (based on its decision to intercept the data message). As thecluster manager stores the network topology of the application cluster,including in which datacenter each other application instance islocated, the cluster manager can easily determine whether thedestination application instance is local (operating in the samedatacenter) or remote (operating in a different datacenter).

When the data message is addressed to an application instance in thesame datacenter, the process 1100 simply forwards (at 1125) the datamessage to the destination application instance without modifying thedata message. In some embodiments, the machine in which the clustermanager operates simply outputs the data message according to itsnetwork stack, or sends the data message to a forwarding element (e.g.,switch or router) to which it connects.

On the other hand, when the data message is addressed to an applicationinstance in a different datacenter, the process 1100 encapsulates (at1115) the data message with tunnel overlay data before forwarding (at1120) the data message to an access point of the remote datacenter. Theencapsulation of some embodiments is based on a tunnel protocol (e.g.,User Datagram Protocol (UDP), Generic Routing Encapsulation (GRE)protocol, Virtual Extensible LAN (VXLAN) protocol, Secure TransactionTechnology (STT) protocol, Secure Shell (SSH) protocol, etc.).

As only certain application instances and their cluster managers in aparticular datacenter are accessible for data messages sent from outsidethe datacenter (e.g., from application instances located in a differentdatacenter), inbound and outbound communication will frequently takedifferent routes through the cluster. FIG. 12 illustrates an example ofcommunications between application instances of a cluster spread acrossdifferent datacenters in four stages 1201-1204. The first stage 1201shows the datacenters 120 and 125 with application instances AI1-AI6, asshown above in FIG. 4.

The first stage 1201 shows that application instance AI2 sends datamessage 1250 destined for a remote application instance AI6. Most remotecommunication will result in one or two hops as data messages are firstforwarded to an access point and then to their target applicationinstance—unless the target is the access point.

The cluster manager (not shown) of application instance AI2 interceptsthe data message 1250 and forwards the data message 1250 to an accesspoint (i.e., the cluster manager for application instance AI4) for thecluster in datacenter 125. In some embodiments, the cluster manager ofapplication instance AI2 selects the access point to which to forwardthe data message 1250 from a list of access points identified for thedatacenter in the cluster topology. In some embodiments, the accesspoint is selected randomly, but, in order to prevent negative effectsfrom out-of-order transmission, a source application instance of someembodiments will only use one access point at any given time or for aparticular group of data messages.

The second stage 1202 shows that the access point cluster managerdetermines that the data message is not intended for applicationinstance AI4 and forwards the packet to the cluster manager for thedestination application instance AI6. As data messages are receivedeither from the network stack or from another cluster manager, thecluster manager determines whether the data messages are destined forthe local application instance or if they require additional forwarding.In some embodiments, data messages are transmitted via UDP as it behavessimilar to raw IP and avoids TCP meltdown. Data messages are only everreturned to the network stack once they reach their final destination insome embodiments.

In the third stage 1203, application instance AI6 receives the datamessage 1250 sends a response data message 1255 back to applicationinstance AI2. Application instance AI6 identifies an access point fordatacenter 120 at AI3 and sends the response data message 1255 to AI3.As noted above, inbound and outbound communication will frequently takedifferent routes, because each cluster manager sends the data message toan access point in the destination datacenter. Because AI2 is not anaccess point, AI6 could not send the response data message back via thesame path (i.e., AI4 would need to send the response data message 1255to an access point in the first datacenter 120 anyway, so AI6 sends thedata message directly to this access point). Finally, the fourth stage1204 shows that application instance AI3 forwards the response packet toapplication instance AI2.

It should be noted that while the example in FIG. 12 only shows twodatacenters, in some cases a cluster might be spread across numerousdatacenters (e.g., 3, 4, 10, etc.). In some cases, one or more of thedatacenters may not have access points exposed to all of the otherdatacenters (e.g., because none of the application instances in aparticular datacenter operate on publicly accessible machines). Forexample, the application instances in a first datacenter might not beable to communicate directly with any of the application instances in asecond datacenter, and vice versa. So long as an access point in a thirddatacenter can communicate with machines in both datacenters, theoverlay network will still work. In this case, a cluster manager for anapplication instance in the first datacenter would forward a datamessage addressed to an application instance in the second datacenter tothe access point in the third datacenter, which would then send the datamessage to an access point in the second datacenter (which subsequentlysends the packet to the cluster manager for the destination applicationinstance). The return path would again be different (assuming the sourceand destination application instances are not located at access points),as the cluster manager for the second datacenter application instancewould send the packet to the access point in the third datacenter, whichsends the packet to an access point in the first datacenter (whichsubsequently sends the packet to the cluster manager for the originalsource application instance).

FIGS. 13 and 14 illustrate examples of processing received data messagesby the cluster manager at an access point. FIG. 13 illustrates anexample of an access point cluster manager processing a received datamessage addressed to an application instance located in the datacenterbut not at the access point in two stages 1301-1302. In this example,machine 720 is a publicly accessible machine, or access point, for adatacenter. As described above, when application instances in onedatacenter send messages destined for application instances in a remotedatacenter, the messages are forwarded by their local cluster managersto an access point (i.e., a publicly accessible machine, clustermanager, application instance, etc.) in the remote datacenter. The firststage 1301 shows that machine 720 receives a data message 1350 atinterface 724. The received data message 1350 has a destination addressfor the application instance AI4, which does not operate on machine 720.In some embodiments, this destination address is part of a tunnelencapsulation on the data message, added to the data message by thecluster manager at the source application instance, as shown in FIG. 10.

The second stage 1302 shows that the cluster manager uses the forwardingtables 738 to determine that the data message 1350 is not addressed to alocal application instance operating on the machine 720. As such, thecluster manager 722 sends the packet back out on the interface 724 fordelivery to the destination application instance through the datacenternetwork.

FIG. 14 illustrates an example of an access point cluster managerprocessing a received data message addressed to an application instancelocated at the access point in two stages 1401-1402. As in the aboveexample, the machine 720 is a publicly accessible access point for thedatacenter. The first stage 1401 shows that machine 720 receives a datamessage 1450 at interface 724. The received data message 1350 has adestination address for the application instance AI2, which operates onthe access point machine 720. In some embodiments, this destinationaddress is part of a tunnel encapsulation on the data message, added tothe data message by the cluster manager at the source applicationinstance, as shown in FIG. 10.

The second stage 1402 shows that the cluster manager uses the forwardingtables 738 to determine that the data message 1450 is addressed to alocal application instance operating on the machine 720. As such, thedecapsulator 1440 decapsulates the data message and delivers this datamessage to the destination application instance AI2 on the machine 720.

FIG. 15 conceptually illustrates a process 1500 of some embodiments forprocessing data messages received at an access point from outside thedatacenter of the access point. In some embodiments, the process 1500 isperformed by a cluster manager that operates on a machine with apublicly accessible address, and therefore serves as an access pointwithin its datacenter for the application instances of a cluster thatare located outside the datacenter.

As shown, the process 1500 begins by receiving (at 1505) a data messagefrom a source application instance. In some embodiments, the datamessage is received from a remote cluster manager, that encapsulated thepacket according to the overlay network and sent the packet through thenetwork between datacenters to the access point machine. In someembodiments, all incoming traffic received through the interface of themachine is sent to the cluster manager, while in other embodiments onlypackets relating to the application cluster overlay are processed by thecluster manager.

The process 1500 then determines (at 1510) whether the data message isaddressed to a local application instance that operates on the machine720 along with the cluster manager. The cluster manager of someembodiments can determine this based on the cluster topology stored forthe one or more clusters to which the applications it manages belong,which indicates both the address of its local application instance aswell as those of the other application instances in the same datacenter.

When the data message is addressed to a local application instance (onthe same machine), the process 1500 sends (at 1515) the data message tothe local network stack (and also decapsulates the data message). Thelocal network stack delivers the data message to the local applicationinstance, which allows the local application instance to process and acton the data message.

On the other hand, when the data message is not addressed to a localapplication instance on the access point machine, the process 1500forwards (at 1520) the data message to the destination instance acrossthe local datacenter. In some embodiments, in order to forward the datamessages, the process 1500 modifies a set of forwarding tables based ona relationship between overlay network addresses and a separate set ofinternal addresses defined in the datacenter. The process 1500 of someembodiments forwards (at 1520) internal data messages (i.e., datamessages destined for application instances in the same datacenter)using the modified set of forwarding tables without encapsulating ordecapsulating the data messages.

IV. Electronic System

Many of the above-described features and applications are implemented assoftware processes that are specified as a set of instructions recordedon a computer readable storage medium (also referred to as computerreadable medium). When these instructions are executed by one or moreprocessing unit(s) (e.g., one or more processors, cores of processors,or other processing units), they cause the processing unit(s) to performthe actions indicated in the instructions. Examples of computer readablemedia include, but are not limited to, CD-ROMs, flash drives, RAM chips,hard drives, EPROMs, etc. The computer readable media does not includecarrier waves and electronic signals passing wirelessly or over wiredconnections.

In this specification, the term “software” is meant to include firmwareresiding in read-only memory or applications stored in magnetic storage,which can be read into memory for processing by a processor. Also, insome embodiments, multiple software inventions can be implemented assub-parts of a larger program while remaining distinct softwareinventions. In some embodiments, multiple software inventions can alsobe implemented as separate programs. Finally, any combination ofseparate programs that together implement a software invention describedhere is within the scope of the invention. In some embodiments, thesoftware programs, when installed to operate on one or more electronicsystems, define one or more specific machine implementations thatexecute and perform the operations of the software programs.

FIG. 16 conceptually illustrates a computer system 1600 with which someembodiments of the invention are implemented. The computer system 1600can be used to implement any of the above-described hosts, controllers,and managers. As such, it can be used to execute any of the abovedescribed processes. This computer system includes various types ofnon-transitory machine readable media and interfaces for various othertypes of machine readable media. Computer system 1600 includes a bus1605, processing unit(s) 1610, a system memory 1625, a read-only memory1630, a permanent storage machine 1635, input machines 1640, and outputmachines 1645.

The bus 1605 collectively represents all system, peripheral, and chipsetbuses that communicatively connect the numerous internal machines of thecomputer system 1600. For instance, the bus 1605 communicativelyconnects the processing unit(s) 1610 with the read-only memory 1630, thesystem memory 1625, and the permanent storage machine 1635.

From these various memory units, the processing unit(s) 1610 retrieveinstructions to execute and data to process in order to execute theprocesses of the invention. The processing unit(s) may be a singleprocessor or a multi-core processor in different embodiments. Theread-only-memory (ROM) 1630 stores static data and instructions that areneeded by the processing unit(s) 1610 and other modules of the computersystem. The permanent storage machine 1635, on the other hand, is aread-and-write memory machine. This machine is a non-volatile memoryunit that stores instructions and data even when the computer system1600 is off. Some embodiments of the invention use a mass-storagemachine (such as a magnetic or optical disk and its corresponding diskdrive) as the permanent storage machine 1635.

Other embodiments use a removable storage machine (such as a floppydisk, flash drive, etc.) as the permanent storage machine. Like thepermanent storage machine 1635, the system memory 1625 is aread-and-write memory machine. However, unlike storage machine 1635, thesystem memory is a volatile read-and-write memory, such a random accessmemory. The system memory stores some of the instructions and data thatthe processor needs at runtime. In some embodiments, the invention'sprocesses are stored in the system memory 1625, the permanent storagemachine 1635, and/or the read-only memory 1630. From these variousmemory units, the processing unit(s) 1610 retrieve instructions toexecute and data to process in order to execute the processes of someembodiments.

The bus 1605 also connects to the input and output machines 1640 and1645. The input machines enable the user to communicate information andselect commands to the computer system. The input machines 1640 includealphanumeric keyboards and pointing machines (also called “cursorcontrol machines”). The output machines 1645 display images generated bythe computer system. The output machines include printers and displaymachines, such as cathode ray tubes (CRT) or liquid crystal displays(LCD). Some embodiments include machines such as a touchscreen thatfunction as both input and output machines.

Finally, as shown in FIG. 16, bus 1605 also couples computer system 1600to a network 1665 through a network adapter (not shown). In this manner,the computer can be a part of a network of computers (such as a localarea network (“LAN”), a wide area network (“WAN”), or an Intranet, or anetwork of networks, such as the Internet. Any or all components ofcomputer system 1600 may be used in conjunction with the invention.

Some embodiments include electronic components, such as microprocessors,storage and memory that store computer program instructions in amachine-readable or computer-readable medium (alternatively referred toas computer-readable storage media, machine-readable media, ormachine-readable storage media). Some examples of such computer-readablemedia include RAM, ROM, read-only compact discs (CD-ROM), recordablecompact discs (CD-R), rewritable compact discs (CD-RW), read-onlydigital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), a varietyof recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.),flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.),magnetic and/or solid state hard drives, read-only and recordableBlu-Ray® discs, ultra density optical discs, any other optical ormagnetic media, and floppy disks. The computer-readable media may storea computer program that is executable by at least one processing unitand includes sets of instructions for performing various operations.Examples of computer programs or computer code include machine code,such as is produced by a compiler, and files including higher-level codethat are executed by a computer, an electronic component, or amicroprocessor using an interpreter.

While the above discussion primarily refers to microprocessor ormulti-core processors that execute software, some embodiments areperformed by one or more integrated circuits, such as applicationspecific integrated circuits (ASICs) or field programmable gate arrays(FPGAs). In some embodiments, such integrated circuits executeinstructions that are stored on the circuit itself.

As used in this specification, the terms “computer”, “server”,“processor”, and “memory” all refer to electronic or other technologicalmachines. These terms exclude people or groups of people. For thepurposes of the specification, the terms display or displaying meansdisplaying on an electronic machine. As used in this specification, theterms “computer readable medium,” “computer readable media,” and“machine readable medium” are entirely restricted to tangible, physicalobjects that store information in a form that is readable by a computer.These terms exclude any wireless signals, wired download signals, andany other ephemeral or transitory signals.

While the invention has been described with reference to numerousspecific details, one of ordinary skill in the art will recognize thatthe invention can be embodied in other specific forms without departingfrom the spirit of the invention. For instance, this specificationrefers throughout to computational and network environments that includevirtual machines (VMs). However, virtual machines are merely one exampleof data compute nodes (DCNs) or data compute end nodes, also referred toas addressable nodes. DCNs may include non-virtualized physical hosts,virtual machines, containers that run on top of a host operating systemwithout the need for a hypervisor or separate operating system, andhypervisor kernel network interface modules.

VMs, in some embodiments, operate with their own guest operating systemson a host using resources of the host virtualized by virtualizationsoftware (e.g., a hypervisor, virtual machine monitor, etc.). The tenant(i.e., the owner of the VM) can choose which applications to operate ontop of the guest operating system. Some containers, on the other hand,are constructs that run on top of a host operating system without theneed for a hypervisor or separate guest operating system. In someembodiments, the host operating system uses name spaces to isolate thecontainers from each other and therefore provides operating-system levelsegregation of the different groups of applications that operate withindifferent containers. This segregation is akin to the VM segregationthat is offered in hypervisor-virtualized environments that virtualizesystem hardware, and thus can be viewed as a form of virtualization thatisolates different groups of applications that operate in differentcontainers. Such containers are more lightweight than VMs.

Hypervisor kernel network interface module, in some embodiments, is anon-VM DCN that includes a network stack with a hypervisor kernelnetwork interface and receive/transmit threads. One example of ahypervisor kernel network interface module is the vmknic module that ispart of the ESXi™ hypervisor of VMware, Inc.

One of ordinary skill in the art will recognize that while thespecification refers to VMs, the examples given could be any type ofDCNs, including physical hosts, VMs, non-VM containers, and hypervisorkernel network interface modules. In fact, the example networks couldinclude combinations of different types of DCNs in some embodiments.

A number of the figures (e.g., FIGS. 4, 5, 8, 11 and 15) conceptuallyillustrate processes. The specific operations of these processes may notbe performed in the exact order shown and described. The specificoperations may not be performed in one continuous series of operations,and different specific operations may be performed in differentembodiments. Furthermore, the process could be implemented using severalsub-processes, or as part of a larger macro process. In view of theforegoing, one of ordinary skill in the art would understand that theinvention is not to be limited by the foregoing illustrative details,but rather is to be defined by the appended claims.

1-24. (canceled)
 25. A method for processing data messages in a firstdatacenter, the method comprising: receiving first and second sets ofdata messages from a first application instance of a cluster ofapplication instances operating on machines located across a pluralityof datacenters including the first datacenter; based on a determinationthat the first set of data messages are destined to an applicationinstance in a second datacenter, encapsulating the data messages in thefirst set of data messages with an overlay network encapsulation headerdefined for the cluster of application instances; and forwarding theencapsulated data message to the other datacenter for forwarding todestination of the first set of data messages; based on a determinationthat the second set of data messages are destined to an applicationinstance in the first datacenter, forwarding the data messages in thesecond set of data messages without encapsulating the data messages withan overlay encapsulation header.
 26. The method of claim 25, wherein acluster manager operating on a machine along with the first applicationinstance performs the encapsulation and forwarding operations.
 27. Themethod of claim 26, wherein the machine is a virtual machine.
 28. Themethod of claim 26, wherein the machine is a physical machine.
 29. Themethod of claim 25, wherein forwarding the encapsulated data messagecomprises forwarding the encapsulated data message to a publiclyaccessible machine of the second datacenter, wherein only a subset ofthe application instances operating in the second datacenter operate onpublicly accessible machines.
 30. The method of claim 25, wherein theoverlay encapsulation header allows the first set of data messages totraverse between the first and second datacenters through an overlaynetwork defined for a cluster of application instances operating in thefirst and second datacenters.
 31. The method of claim 25 furthercomprising maintaining a cluster topology of the cluster of applicationinstances that identifies, for each application instance of the cluster,the datacenter in which the application instance operates.
 32. Themethod of claim 25, wherein encapsulating the first set of data messagescomprises using encapsulation headers associated with a tunnelingprotocol.
 33. The method of claim 25, wherein the encapsulation headerstores an address associated with the second application instance thatis an overlay address assigned to the second application instance in theoverlay network.
 34. The method of claim 25, wherein the applicationinstances of the cluster form a complete graph in an overlay network,wherein each application instance is connected to every otherapplication instance in the cluster.
 35. A non-transitory machinereadable medium storing a program for processing data messages in afirst datacenter, the program for execution by at least one processingunit, the program comprising sets of instructions for: receiving firstand second sets of data messages from a first application instance of acluster of application instances operating on machines located across aplurality of datacenters including the first datacenter; based on adetermination that the first set of data messages are destined to anapplication instance in a second datacenter, encapsulating the datamessages in the first set of data messages with an overlay networkencapsulation header defined for the cluster of application instances;and forwarding the encapsulated data message to the other datacenter forforwarding to destination of the first set of data messages; based on adetermination that the second set of data messages are destined to anapplication instance in the first datacenter, forwarding the datamessages in the second set of data messages without encapsulating thedata messages with an overlay encapsulation header.
 36. Thenon-transitory machine readable medium of claim 35, wherein a clustermanager operating on a machine along with the first application instanceperforms the encapsulation and forwarding operations.
 37. Thenon-transitory machine readable medium of claim 36, wherein the machineis a virtual machine.
 38. The non-transitory machine readable medium ofclaim 36, wherein the machine is a physical machine.
 39. Thenon-transitory machine readable medium of claim 35, wherein the set ofinstructions for forwarding the encapsulated data message comprises aset of instructions for forwarding the encapsulated data message to apublicly accessible machine of the second datacenter, wherein only asubset of the application instances operating in the second datacenteroperate on publicly accessible machines.
 40. The non-transitory machinereadable medium of claim 35, wherein the overlay encapsulation headerallows the first set of data messages to traverse between the first andsecond datacenters through an overlay network defined for a cluster ofapplication instances operating in the first and second datacenters. 41.The non-transitory machine readable medium of claim 35, the programfurther comprising a set of instructions for maintaining a clustertopology of the cluster of application instances that identifies, foreach application instance of the cluster, the datacenter in which theapplication instance operates.
 42. The non-transitory machine readablemedium of claim 35, wherein the set of instructions for encapsulatingthe first set of data messages comprises a set of instructions for usingencapsulation headers associated with a tunneling protocol.
 43. Thenon-transitory machine readable medium of claim 35, wherein theencapsulation header stores an address associated with the secondapplication instance that is an overlay address assigned to the secondapplication instance in the overlay network.
 44. The non-transitorymachine readable medium of claim 35, wherein the application instancesof the cluster form a complete graph in an overlay network, wherein eachapplication instance is connected to every other application instance inthe cluster.